Air Paradis : Detect bad buzz with deep learning

Context

"Air Paradis" is an airline company who's marketing department wants to be able to detect quickly "bad buzz" on social networks, to be able to anticipate and address issues as fast as possible. They need an AI API that can detect "bad buzz" and predict the reason for it.

The goal here is to evaluate three approaches to detect "bad buzz" :

• cloud managed service : Azure Cognitive Service for Language - Sentiment analysis
• simple model : Logistic Regression trained on pre-processed data (stopwords, stemming, lemmatization, ...)
• advanced models : deep learning models (Keras)
  • with word embedding (Gensim : word2vec, Glove, fasttext)
  • with a "Long short-term memory" (LSTM) layer
  • with a Bidirectional Encoder Representations from Transformers (BERT)

Load project modules

The helpers functions and project specific code will be placed in ../src/.

We will use the Python programming language, and present here the code and results in this Notebook JupyterLab file.

We will use the usual libraries for data exploration, modeling and visualisation :

• NumPy and Pandas : for maths (stats, algebra, ...) and large data manipulation
• scikit-learn : for machine learning models training and evaluation
• Plotly : for interactive data visualization

We will also use libraries specific to the goals of this project :

• NLP Natural Language Processing
  • NLTK and Spacy : for text processing
  • Gensim and pyLDAvis : for topic modelling and visualisation
  • Kereas : for deep learning models training and evaluation

import pickle

# Import custom helper libraries
import os
import sys

src_path = os.path.abspath(os.path.join("../src"))
if src_path not in sys.path:
    sys.path.append(src_path)

import features.helpers as feat_helpers
import data.helpers as data_helpers
import visualization.helpers as viz_helpers

# Maths modules
from scipy.stats import f_oneway
import pandas as pd

# Viz modules
import plotly.express as px

# Render for export
import plotly.io as pio

pio.renderers.default = "notebook"

Exploratory data analysis (EDA)

We are going to load the data and analyse the distribution of each variable.

Load data

Let's download the data from the Kaggle - Sentiment140 dataset with 1.6 million tweets competition.

# Download and unzip CSV files
!cd .. && make dataset && cd notebooks

>>> Downloading and extracting data files...
Data files already downloaded.
>>> OK.

Now we can load the data.

# Load data from CSV
df = pd.read_csv(
    os.path.join(
        "..", "data", "raw", "training.1600000.processed.noemoticon.csv"
    ),
    names=["target", "id", "date", "flag", "user", "text"],
)

# Reduce memory usage
df = data_helpers.reduce_dataframe_memory_usage(df)

Explore data

Let's display a few examples, find out how many data points are available, what are the variables and what is their distribution.

# Display first few rows
df.head(5)

	target	id	date	flag	user	text
0	0	1467810369	Mon Apr 06 22:19:45 PDT 2009	NO_QUERY	_TheSpecialOne_	@switchfoot http://twitpic.com/2y1zl - Awww, t...
1	0	1467810672	Mon Apr 06 22:19:49 PDT 2009	NO_QUERY	scotthamilton	is upset that he can't update his Facebook by ...
2	0	1467810917	Mon Apr 06 22:19:53 PDT 2009	NO_QUERY	mattycus	@Kenichan I dived many times for the ball. Man...
3	0	1467811184	Mon Apr 06 22:19:57 PDT 2009	NO_QUERY	ElleCTF	my whole body feels itchy and like its on fire
4	0	1467811193	Mon Apr 06 22:19:57 PDT 2009	NO_QUERY	Karoli	@nationwideclass no, it's not behaving at all....

# Diaplay number of rows and colmn types
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1600000 entries, 0 to 1599999
Data columns (total 6 columns):
 #   Column  Non-Null Count    Dtype   
---  ------  --------------    -----   
 0   target  1600000 non-null  Int8    
 1   id      1600000 non-null  UInt32  
 2   date    1600000 non-null  string  
 3   flag    1600000 non-null  category
 4   user    1600000 non-null  string  
 5   text    1600000 non-null  string  
dtypes: Int8(1), UInt32(1), category(1), string(3)
memory usage: 48.8 MB

There are 1600000 rows, each composed of 6 columns :

• target: the polarity of the tweet (0 = negative, 2 = neutral, 4 = positive)
• id: The id of the tweet ( 2087)
• date: the date of the tweet (Sat May 16 23:58:44 UTC 2009)
• flag: The query (lyx). If there is no query, then this value is NO_QUERY.
• user: the user that tweeted (robotickilldozr)
• text: the text of the tweet (Lyx is cool)

We are only interrested in the target and text variables. The rest of the columns are not useful for our analysis.

# Drop useless columns
df.drop(columns=["id", "date", "flag", "user"], inplace=True)

# Replace target values with labels
df.target.replace(
    {
        0: "NEGATIVE",
        2: "NEUTRAL",
        4: "POSITIVE",
    },
    inplace=True,
)

# Display basic statistics
df.describe(include="all")

	target	text
count	1600000	1600000
unique	2	1581466
top	NEGATIVE	isPlayer Has Died! Sorry
freq	800000	210

# Plot target distribution
viz_helpers.histogram(
    df, label_x="target", label_colour="target", title="Target distribution"
)

There are exactly as many (800000) POSITIVE tweets as NEGATIVE tweets. There are no NEUTRAL tweets. The problem is well balanced and there will be no bias towards one class during the training of our models.

# Plot text length distribution
df["text_length"] = df.text.str.len()

p_value = f_oneway(
    df.loc[df["target"] == "NEGATIVE", "text_length"],
    df.loc[df["target"] == "POSITIVE", "text_length"],
)[1]

viz_helpers.histogram(
    df,
    label_x="text_length",
    label_colour="target",
    title=f"Text length distribution / p-value={p_value:.5f}",
    include_boxplot=True,
)

There are no big difference between the POSITIVE and NEGATIVE tweets, but NEGATIVE tweets are slightly longer than POSITIVE tweets. In both classes, there are two modes : ~45 characters and 138 characters (the maximum allowed at some point).

# Plot word count distribution
df["word_count"] = df.text.str.split().str.len()

p_value = f_oneway(
    df.loc[df["target"] == "NEGATIVE", "word_count"],
    df.loc[df["target"] == "POSITIVE", "word_count"],
)[1]

viz_helpers.histogram(
    df,
    label_x="word_count",
    label_colour="target",
    title=f"Word count distribution / p-value={p_value:.5f}",
    include_boxplot=True,
)

There are no big difference between the POSITIVE and NEGATIVE tweets, but NEGATIVE tweets are significatively longer than POSITIVE tweets. In both classes, there are two modes : ~7 words and ~20 words.

Text analysis

We will look more in details at what contains the text variable.

First, we will transform the dataset into a Bag of Words representation with TfIdf (Term Frequency - Inverse Document Frequency) weights. To achieve this, we are going to use th SpaCy tokenizer.

# Vectorizers
from sklearn.feature_extraction.text import TfidfVectorizer

# Tokenizers, Stemmers and Lemmatizers
import nltk
from nltk.corpus import stopwords
import spacy

# Download resources
nltk.download("stopwords")
stopwords = set(stopwords.words("english"))

# Download SpaCy model
try:
    nlp = spacy.load("en_core_web_sm")
except:
    !python -m spacy download en_core_web_sm
    nlp = spacy.load("en_core_web_sm")

# Define tokenizer
tokenizer = lambda text: [  # SpaCy Lemmatizer
    token.lemma_.lower()
    for token in nlp(text)
    if token.is_alpha and not token.is_stop
]

2022-01-31 11:19:06.217345: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-01-31 11:19:06.217391: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
[nltk_data] Downloading package stopwords to
[nltk_data]     /home/clement/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!

# Processed data path
processed_data_path = os.path.join("..", "data", "processed")
vectorized_dataset_file_path = os.path.join(processed_data_path, "tfidf_spacy_dataset.pkl")
vocabulary_file_path = os.path.join(
    processed_data_path, "tfidf_spacy_vocabulary.pkl"
)

if os.path.exists(vectorized_dataset_file_path) and os.path.exists(
    vocabulary_file_path
):
    # Load vectorized dataset
    with (open(vectorized_dataset_file_path, "rb")) as f:
        X = pickle.load(f)
    # Load vocabulary
    with (open(vocabulary_file_path, "rb")) as f:
        vocabulary = pickle.load(f)
else:
    # Define vectorizer
    vectorizer = TfidfVectorizer(
        strip_accents="unicode",
        lowercase=True,
        stop_words=stopwords,
        tokenizer=tokenizer,
    )

    # Vectorize text
    X = vectorizer.fit_transform(df.text)

    # Get vocabulary
    vocabulary = vectorizer.get_feature_names_out()

    # Save vectorized dataset as pickle
    with open(vectorized_dataset_file_path, "wb") as f:
        pickle.dump(X, f)

    # Save vocabulary as pickle
    with open(vocabulary_file_path, "wb") as f:
        pickle.dump(vocabulary, f)

Our corpus is now transformed into a BoW representation. We can analyse the words frequencies.

# List words TF-IDF scores
words = pd.Series(X.sum(axis=0).A1, index=vocabulary)

# Top 20 tokens by TfIdf
top_20_words = words.nlargest(20).sort_values(ascending=False)

# Plot top 20 tokens by TfIdf
fig = px.bar(
    top_20_words,
    x=top_20_words.index,
    y=top_20_words.values,
    labels={"x": "Words", "y": "Count", "color": "Count"},
    title=f"Top 20 important words (Tf-Idf) - Vocalbulary size: {len(vocabulary)}",
    color=top_20_words.values,
)
fig.show()

We can see that the most important words actually meaningful and relevant regarding the sentiment associated to each message.

We will use this representation to build a classification model in order to predict the sentiment of a new message.

Classification model

We are going to train and evaluate a classification model to predict the sentiment of a new message.

Dimension reduction & Topic modeling

First, we need to reduce the dimension of the BoW representation : there are 240589 words in the corpus. We use the Latent Semantic Analysis (LSA) method to reduce the dimension of the BoW representation. This method will create Topics that are the most relevant to the corpus.

from sklearn.decomposition import TruncatedSVD


# Train LSA model
n_components = 50
lsa = TruncatedSVD(n_components=n_components, random_state=42).fit(X)

# Plot explained variance ratio of LSA
fig = px.line(
    x=range(1, n_components + 1),
    y=lsa.explained_variance_ratio_,
    title="Explained variance ratio of LSA",
    labels={"x": "Component", "y": "Explained variance ratio"},
    markers=True,
)
fig.show()

The elbow method should help us to choose the number of topics, but here there is no clear elbow, so we choose 50 topics.

# Reduce dimensionality
X_lsa = lsa.transform(X)

X_lsa.shape

(1600000, 50)

The dataset is now reduced to 50 topics. We can observe the composition (most relevant words) of each topic.

# Plot the top words of each topic
viz_helpers.plot_top_words(
    model=lsa,
    feature_names=vocabulary,
    n_top_words=10,
    n_topics=50,
    title=f"LSA Topics / n_components={n_components}",
)

We can identify the following topics :

• "sleep" related : topics 8, 14, 17, 18, 46, ...
• "movie" related : topics 25, 30, ...
• "work" related : topics 1, 3, 4, ...
• ...

Train and test the model

We can now train and test a classification model. We are going to use the Logistic Regression model.

from sklearn.model_selection import train_test_split


# Train-test split
X_train, X_test, y_train, y_test = train_test_split(
    X_lsa,
    df.target,
    test_size=0.2,
    stratify=df.target,
    random_state=42,
)

from sklearn.linear_model import LogisticRegressionCV


# Define model
model = LogisticRegressionCV(random_state=42)

# Train model
model.fit(X_train, y_train)

LogisticRegressionCV(random_state=42)

Once the model is trained, we can observe which topics are the most relevant to the sentiment of the messages.

# Comute the coefficients
topics = [f"Topic {i + 1}" for i in range(n_components)]
coefs = pd.Series(
    model.coef_[0],
    index=topics,
)

# Top 20 topics by importance (positive and negative)
top_20_coefs = (
    coefs.nlargest(10).append(coefs.nsmallest(10)).sort_values()
)

# Plot top 20 topics by importance (positive and negative)
fig = px.bar(
    top_20_coefs,
    x=top_20_coefs.index,
    y=top_20_coefs.values,
    labels={"x": "Topic", "y": "Importance", "color": "Importance"},
    title=f"Top 20 important topics",
    color=top_20_coefs.values,
)
fig.show()

The most NEGATIVE topics are :

• 25 : "sad", "watch", "movie", ...
• 28 : "sorry", "sad", "hear", ...
• 6 : "miss", ...

The most POSITIVE topics are :

• 2 : "thank", ...
• 24 : "night", "great", "hop", "morning", "fun", ...
• 7 : "love", ...

Now we can measure the performance of our model. We are going to use the Confusion Matrix, the Precision-Recall curve (Average Precision metric) and the ROC curve (ROC AUC metric) to evaluate our model.

viz_helpers.plot_classifier_results(
    model,
    X_train,
    y_train,
    title="Train set results",
)

viz_helpers.plot_classifier_results(
    model,
    X_test,
    y_test,
    title="Test set results",
)

The performances on the train and test datasets are identical, so we know our model is well trained (no over/under-fitting).

The performances are quite correct for a baseline model :

• Average Precision = 0.73
• ROC AUC = 0.74

Our model is biased towards the POSITIVE class : it predicted 35% more POSITIVE (918049) messages than NEGATIVE (681951).

Let's observe some classification errors.

# Compute predictions
y_pred = model.predict(X_lsa)
df["prediction"] = y_pred

import shap

shap.initjs()

explainer = shap.Explainer(model, X_train, feature_names=topics)
shap_values = explainer(X_lsa)

Linear explainer: 1600001it [00:16, 34368.14it/s]                             

# False positive example
fp_index = df[(df.target == "NEGATIVE") & (df.prediction == "POSITIVE")].index[0]
fp_text = df.text.values[fp_index]

print(fp_text)

shap.plots.force(shap_values[fp_index])

@switchfoot http://twitpic.com/2y1zl - Awww, that's a bummer.  You shoulda got David Carr of Third Day to do it. ;D

Visualization omitted, Javascript library not loaded!
Have you run `initjs()` in this notebook? If this notebook was from another user you must also trust this notebook (File -> Trust notebook). If you are viewing this notebook on github the Javascript has been stripped for security. If you are using JupyterLab this error is because a JupyterLab extension has not yet been written.

On this false-positive example, we can see that the model is not able to predict the sentiment of the message, despite words like "bummer" and "should"...

# False negative example
fn_index = df[(df.target == "POSITIVE") & (df.prediction == "NEGATIVE")].index[0]
fn_text = df.text.values[fn_index]

print(fn_text)

shap.plots.force(shap_values[fn_index])

Being sick can be really cheap when it hurts too much to eat real food  Plus, your friends make you soup

Visualization omitted, Javascript library not loaded!
Have you run `initjs()` in this notebook? If this notebook was from another user you must also trust this notebook (File -> Trust notebook). If you are viewing this notebook on github the Javascript has been stripped for security. If you are using JupyterLab this error is because a JupyterLab extension has not yet been written.

On this false-negative example, we can see that the model is not able to predict the sentiment of the message. But in this case, the model is fooled by the presence of words like "sick", "cheap", "hurts", ...